Discovering Cyclic Causal Models in Psychological Reesearch

Network Theory in Psychology

Mental disorder is produced by direct causal interactions between symptoms that reinforce each other via feedback loops. (Borsboom & Cramer, 2013).

Inferring causality from observational data

And what has happened a lot in psychology is that many empirical researchers try to gain insights into these causal relationships by fitting this statistical network model using observational data. [Alt + click!] But these connections in the network are merely statistical relations, which cannot be directly translated into causal relations.

And more and more people have become aware of this issue, and there has come another popular casual modeling tool, which is DAG. DAG stands for directed acyclic graphical model. It has all these arrows you can interpret as a cause and effect. [Alt + click!] It seems very promising, however, one of biggest shortcomings of DAG is by definition it does not allow cycles. But as we saw before in the previous slide, the true system of psychopathology we are interested in, contains cycles. And in fact, the cycles are critical in understanding this central dynamic of symptoms. In that sense, we cannot really use DAG to estimate the real underlying mechanism. So what do we need to do?

Directed Acyclic Graph (DAG)

Directed Cyclic Graph (DCG)

Problem: Estimating a cyclic causal model is fundamentally very difficult. Relaxing the acyclicity assumption entails much of theoretical complication.

Solutions exist

Cyclic Causal Discovery (CCD) (Richardson, 1996)
Fast Causal Inference (FCI) (M. Mooij & Claassen, 2020)
Cyclic Causal Inference (CCI) (Strobl, 2019)

Then we see the results right here. First looking at the structural hamming distance, we see that in general CCI the green line and FCI the yellow line perform better in sparse conditions while CCD the red lines performs better in dense conditions. Remember this is structural hamming distance, so the lower the better. the less steps it takes the better estimated graph it is. Another thing to note i that it is a bit strange in 5p dense conditions. We see a bit of dips with the small sample sizes and actually it increases as sample size gets bigger… But for now let’s move on.

Then we see precision. Here we see that FCI, the yellow one, actually doesn’t do so well. it records almost always the low precision. And also very similar pattern is observed with recall values.

Then we look at the uncertainty rate. Then we see CCD actually produces relatively many more circle endpoints than the others. Except for the rather strange 5p dense conditions, the CCD almost always records the highest uncertainty rate.

Well, what does it mean? It is not so straightforward what is happening here. So we look at the actual graphs.

Summary

Causal inference is the fundamental interest in science.
The underlying dynamic processes of many systems contain cycles.
Our study showcases the cyclic causal discovery algorithms that are potentially suitable for typical psychological observational data.
Causal discovery methods could provide much richer insights into the underlying causal dynamics of the system than statistical network models.
Conclusion is rather nuanced (no one-size-fits-all algorithm).
Learning cyclic causal models from observational data is challenging.

To sum it up, causal inference is the fundamental interest in science regardless of field of study.

And often, the underlying causal mechanism of interest contains cycles.

In this project, we show some of the possibilities to researchers that there are methods available that can estimate the cyclic causal models.

And in fact they can provide all these causal directions and some algorithms can even flag the possibility of latent confounder which are more insightful than using statistical network models.

But, we cannot say one specific algorithm outperforms in all different scenarios. And unfortunately our conclusion is rather nuanced. We believe that there is no single algorithm that is suitable for all scenarios. Researchers must consider the characteristics of the causal system of interest and relative importance of different aspects in their research questions.

As a conclusion, estimating causal model with cycles based on observational data is very very challenging and it is not too super easy to understand how they work and how to interpret their output sometimes. There are much more to be done in this topic, but I do hope that the current project would encourage researchers to look at the cyclic causal discovery methods.

References

Borsboom, D., & Cramer, A. O. J. (2013). Network analysis: An integrative approach to the structure of psychopathology. Annual Review of Clinical Psychology, 9(1), 91–121. https://doi.org/10.1146/annurev-clinpsy-050212-185608

M. Mooij, J., & Claassen, T. (2020). Constraint-based causal discovery using partial ancestral graphs in the presence of cycles. In J. Peters & D. Sontag (Eds.), Proceedings of the 36th conference on uncertainty in artificial intelligence (UAI) (Vol. 124, pp. 1159–1168). PMLR. https://proceedings.mlr.press/v124/m-mooij20a.html

Richardson, T. (1996). Discovering cyclic causal structure. Carnegie Mellon [Department of Philosophy].

Strobl, E. V. (2019). A constraint-based algorithm for causal discovery with cycles, latent variables and selection bias. International Journal of Data Science and Analytics, 8(1), 33–56. https://doi.org/10.1007/s41060-018-0158-2

Theoretical complication

Global Markov property is no longer guaranteed.
- Need extra restrictions on \(P\) (e.g., linearly independent error terms (\(\varepsilon\)))
Cyclic model is not always statistically identified (even in linear case).
- many equivalent models !
Equilibrium state is necessary (All \(|\lambda| < 1\)).

There are many problems with assumptions when estimating causal model with cycles. But I picked three the most crucial ones in my opinion. So when it comes to cyclic models, the most crucial issues are first, the global markov property which is the necessary property in order to read off the graphical model, that is often violated in the models with cycles.
And another major issue is that the cyclic models are often not statistically identified, meaning that either there are too many solutions possible or no solution at all.
Lastly, not all cycles can be estimated. The ones we can estimate are limited such that it has to converge to some equilibrium states, and in order for that to happen, all Eigen values of regression matrix has to be smaller than 1 in absolute values!

So this causes quite some headaches when trying to estimate the cyclic models.

Solutions exist, but…

Project goals

Give an accessible overview of the algorithms.
Investigate the performance of each algorithm.
Apply to empirical data to assess practical applicability.

Simulation settings

So this is the simulation settings. We vary the number of edges, so the density of the model, as you see in the columns, and the size of the models, so the number variables, you see that in the rows Also, we differ the conditions by adding the latent confounder, which is represented as L1 and L2 in the figure. Lastly, we also vary the sample sizes ranging from 50 to 10,000. In the end, we have 2 different densities, 2 different model sizes, and either absence or presence of confounder, and 10 different samples. So that lead us to 2 by 2 by 2 by 10 simulation design. And for each of this different condition, we run all three algorithms 500 times, So that is 80 conditions times by 3 algorithms times by 500 iterations. So in the end we estimate 120,000 different graphs in our simulation. not really important number just give you a idea.

Constraint-based algorithm output

CPDAG
CCD PAG
FCI PAG
CCI PAG

Performance evaluation

So what do we then compare or evaluate the algorithms? The basic idea is that, we have this true DCG, so directed cyclic graph. And we construct the true ancestral graph corresponding to this underlying DCG. Why do we do that… Well, it’s because the output of these algorithms are called PAG that I briefly described earlier. So these cyclic causal discovery algorithms fundamentally cannot estimate the true underlying directed cyclic graph, but they aim to estimate this ancestral graph. This is definitely due to the challenges coming from the cyclicity nature. We can delve deeper into this if anyone is curious. But for now, remember that we are here aiming to recover the true ancestral graph.

So then, basically we assess the performance of each algorithm by comparing their outputs, the PAGs to the true ancestral graph. so how well they can retrieve the true model?

Evaluation metrics

Structural Hamming distance (SHD) = \(A\) (Addition) + \(D\) (Deletion) + \(C\) (Changes)
Precision = \(\frac{TP} {(TP + FP)}\)
Recall = \(\frac{TP} {(TP + FN)}\)
Uncertainty rate = \(\frac{\text{Number of circle endpoints} (\circ)}{\text{Total number of possible endpoints}}\)

Example:
- For the arrow head (\(>\)): precision = \(\frac{4}{4 + 3 + 0}\) and recall = \(\frac{4}{4 + 0 + 0}\).
- The value of SHD for the example PAG output from (b) – provided that the true ancestral graph is (a) – is 6: 0 (A) + 0 (D) + 6 (C).

Well, how do we compare exactly? So these are evaluation metrics we use in our simulation. We first look at the structural hamming distance as an overall performance. So this more of a global evaluation metric. And we also look at three different local metrics, which are preicision, recall, and uncertainty rate. They are defined as such, which is not immeidiately intuitive.

But these are actually very simple. So this is a simple example. Let’s say (a) is the true ancestral graph. and (b) is the estimated graph. Then first, we can compute the structural hamming distance, the name sounds a bit scary but it is just a simply summing all the steps we need going from one graph to the other. So in this case, how many steps do we need from the estimated graph (b) to the true graph (a): first we see that there is no edge needed to be added nor deleted. Then we see, we need to change six edge endpoints. therefore, the structural hamming distance in this case is six. Then moving on to the precision, recall, and uncertainty rate. This is the confusion matrix for this example; we have three different categories: the arrow head, arrow tail, and NULL which means there is no edge at all. One thing to note here is that we do not count the circle endpoint when computing precision or recall! This is pretty important point for later. Anyways, we can for example, compute the precision and recall for the arrow head. just simply following definition, precision is 4 over 7 and recall is 4 over 4, so 1 in this case. Lastly, uncertainty rate is again very simple. We just compute the proportion of circle endpoints. So in this case we have 3 circles so that gives us 3 over 20! where does 20 come from? 20 is the total possible edge endpoint in this graph. computed by 5 choose 2 times by 2 which is 20. So this was a long explanation on how we evaluated the performance of the algorithm.

Typical behavior of algorithms

(a): True ancestral graph
(b): PAG estimated by CCD
(c): PAG estimated by FCI
(d): PAG estimated by CCI

So these are the estimated graphs from each algorithm for the most simple condition, 5p sparse case. (a) is the true graph. and (b) is estimated by CCD, (c) is estimated by FCI, and (d) is estimated by CCI.

What is typically happening with these algorithm is basically that CCD tends to produce more circle but the directions it predicts tend to be all correct. On the other hand, FCI barely outputs any circle but it rather guesses quickly. But the directions are not very accurate. Lastly, CCI produces some circles but the directions are also likely to be correct. So this is what usually happens. CCD tends to be more conservative, and FCI is just guessing all the time (and often not correctly), while CCI is less conservative than CCD but it mostly gets the directions correct.

So that was our general observation from the simulation study.

Empirical example

Then we look at what happens when we apply them to an empirical data. This was psychological data where they measured depression symptoms from about 400 people, cross-sectional data. So (a) is the standard statistical network model, often called Gaussian graphical model. and (b) is the PAG estimated by CCD. (c) is the PAG estimated by CCI. and (d) is the PAG estimated by FCI.

Well, in this case, we don’t really know what is the underlying true causal structure. But just by comparing the results, we could say that the structures themselves do not look so different. They all kinda show this three-clusters. But then causal graphs are much more sparse than statistical network model, which implies that it is probably not a good idea even to use network model as a guide to the causal skeleton.

Overall, this example highlights that the PAGs recovered by cyclic causal discovery algorithms provide more insights into this possible causal relationships that cannot be obtained from statistical network model alone.

Also, we saw that different algorithms appeared to agree on many features of the causal structure but of course they show some different inference on causal directions. Then of course your question could be, which one should i believe? well… that’s a question that I cannot just answer you. Which algorithm you should go for, depends on multiple factors like what assumptions do you have in your causal structure of interest, like do you believe that there are a lot of latent confounders or do you believe the causal relationships are linear… because all these algorithm make somewhat different assumptions as i showed earlier. So you need to think about that. And also, based on our simulation results, you can also consider what your priority is in terms of your research question. Do you care more about not getting wrong inferences or do you care more about getting the most inferences at the expense of getting some directions wrong, then you might prefer CCI. So it depends, and given the information that I’ve gotten so far. my advice, or my conclusion is it needs some more work, actually a lot of more work to do, and for now, there seems to be one single algorithm that performs well in every scenario.

Possible practical application

Personalized psychotherapy (target symptoms)
Medical: effective treatment design

Q: “what are the potential application of this project, so what does this mean to a clinical psychologist?”

For example, a clinical psychologist could use this to customize the therapy based on the found causal dynamics. Let’s imagine that this is a person’s real depression model with cycles, then we can see that insomnia is the central node that has a lot of connections causing other depression symptoms. So we can effectively treat this patient’s depression by targeting the sleeping problems.
I think this would work more or less the same way to other medical treatments. You can imagine that this is a cyclic causal model for some other disease, right. Then, we can also design a more effective treatment once we are able to identify which is the most influential cause of the disease. So yeah this would be one of the crucial clinical implications of this project.

Follow-up

Possible combination with different types of causal discovery algorithm. \(\rightarrow\) Hybrid!

CCD+ GES (greedy equivalence search)

Q:“What could be an interesting follow-up or extension you could do if more fundings become available?”

So I just told you that the algorithm usually cannot identify the one true graph but instead give you this whole set. And that’s the general limitations of the causal discovery algorithms, that they often don’t give you the full picture.

So let’s say this is the true model.And usually, the algorithms can only identify this up to a certain extent. So you see these edges are left undirected. Then I think we can use another algorithm, to orient the rest of edges.

For example, we could combine CCD with GES, greedy equivalence search algorithm, which utilizes different types of information to find an optimal model.

So like this, we can combine these two and get the full picture.

This extension is very plausible in my opinion and promising but it’s not known yet, which combination would work the best in practice. So this would be a great work to do if there is extra resources available in the future.

Thank you